Research is reproducible when the data and the code used to obtain a finding are available and sufficient for an independent researcher to recreate the finding.

• computational, data-intensive

• spans the full data, analysis, & publication workflow

• most of us have received only perfunctory training (if any)

Victoria Stodden, F. Leisch, & R. Peng, ed., Implementing Reproducible Research, CRC Press, 2014.
Christopher Gandrud, Reproducible Research with R and RStudio, 2/e, CRC Press, 2015.

Events tell us:

More accountability is needed because of

• data falsification
• erroneous analysis
• misleading presentation of results

Karen EC Levy & David Merritt Johns, When open data is a Trojan Horse: The weaponization of transparency in science and governance, Big Data and Society, 2016.

the primary findings were false. The major effect disappeared after correcting for

• coding errors

• selective exclusion of available data

• unconventional weighting of summary statistics

Kenneth Rogoff & Carmen Reinhart

data were falsified to obtain the research outcomes he wanted, resulting in

• retracted journal articles (11 to date)

• terminated clinical trials

• cancelled research funding

• civil suit by patients

Anil Potti

Scientists and skeptics are in a knife fight, and you don’t bring data to a knife fight.
— Paul Erlich

Why should I make the data available to you, when your aim is to try and find something wrong with it?
— Phil Jones

1000 years of temperature variation: the ”hockey stick” graph by Michael Mann

The primary benficiary is you

If you do anything “by hand”" once, you’ll do it 100 times.

— Paul Wilson, UW–Madison

Your closest collaborator is you, six months ago. Have you tried to email that slacker?

— Karl Broman, UW–Madison

To preserve sanity, stop collaborating via email, attachments, and tracking changes in Word.

— Jenny Bryan, UBC

• Write scripts (avoid manual copy, paste, mouse-clicks)

• Plan the organization and naming scheme for files

• Strive for simplicity, readability, reusability, and testability

• Agree on a workflow for collaborating before starting a manuscript

• DRY (don’t repeat yourself)

• Link files explicitly

• Plan data management

• Use version control

• Postpone optimization

• License your software

• Write scripts (avoid manual copy, paste, mouse-clicks)

• Plan the organization and naming scheme for files

• Strive for simplicity, readability, reusability, and testability

• Agree on a workflow for collaborating before starting a manuscript

• DRY (don’t repeat yourself)

• Link files explicitly

• Plan data management

• Use version control

• Postpone optimization

• License your software

Consider a sample report

Imagine that you were the author of the “Load cell calibration report”

Carefully review the report and answer Mystery question 2:

Identify as many “manual operations”
as possible.